Methods and apparatus for generating a continuum of image data

ABSTRACT

The present invention provides methods and apparatus for generating a continuum of image data. The continuum can include image data representing a street level view of a geographic area. The image data can be captured at multiple disparate points along another continuum. Each continuum of image data can include a ribbon of data representing the geographic area. In some examples, image data can be simultaneously captured which represents multiple continuums of image data.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims priority to pending patent applicationSer. No. 11/216,465, filed, Aug. 31, 2005 and entitled, “Apparatus andMethod for Producing Video Drive-By Data Corresponding to a GeographicLocation,” the contents of which are relied upon and incorporated byreference, and also the co-pending patent application filed concurrentlywith the present invention and entitled, “Methods and Apparatus forPresenting a Continuum of Image Data.”

FIELD OF THE INVENTION

The present invention relates to methods and apparatus of generatingimage data. More specifically, the present invention relates togenerating data descriptive of a continuum of images, such as thosecomprising a geographic setting.

BACKGROUND OF THE INVENTION

Images of geographic areas created on a local basis have been created innumerous formats. Photographs, movie cameras, video camera recorders,and more recently digital recorders have all been utilized to capturemoving images of a geographic parcel. Photographs can be useful becausethey are easily associated with a particular real estate parcel, howeverthey are limited in the scope of the subject matter that they caninclude. In addition, photographs must be manually acquired and docketedin order to be associated with a property and subsequently retrieved.Panoramas can increase the scope of the subject matter to a point, butare limited by a point of capture of the panoramic views.

Movie pictures, video recorders and digital recorders and other “motionpictures” provide for increased scope of image capture. However, it issometimes difficult to locate and view a particular portion of interestof images captured as motion pictures. In addition, correlation ofparticular portions of a motion picture with a particular real estateparcel can be difficult if the motion picture includes more than onereal estate parcel. For example, it is relatively difficult to locateand view a particular real estate parcel in the context of itsneighborhood setting, or particular aspects of its setting.

In addition, aerial images, such as satellite pictures of geographicareas have been captured and specific parcels of land or landmarks canbe located on the aerial images.

However, prior to the present invention, there has not been a mechanismfor accurately correlating ground level images with substantial portionsof an aerial image in a consistent and orderly format that allows forthe identification of a particular parcel of land and provide bothaerial and ground level views of the parcel, as well as a surroundingarea of the parcel. Similarly, prior to the present invention, there hasnot been a method for correlating geopolitical indicators, such asproperty lot lines, or tax map parcels with aerial images and groundlevel video images.

Substantial portions of ground level images have not been correlatedwith aerial images or with geopolitical indicators, in part, becausethere has not been any image vehicle capable to providing a format ofhigh quality and accurate representations of street level images capableof being matched to macro level image data.

SUMMARY OF THE INVENTION

Accordingly, the present invention provides methods and apparatus forgenerating a continuum of image data. In some embodiments, the continuumof image data provides a two dimensional ribbon of street level views ofgeographic areas. The ribbons are created by capturing two or moreimages of a subject, wherein each of the two or more images are capturedfrom disparate points on a continuum. For example, each of the two ormore images can be captured from a different position on a street as avehicle drives down the street. Generally, the images will be capturedfrom an orthogonal view of the subject.

Portions of the two or more images are then aligned in a dimensionconsistent with the continuum, wherein, according to our example thecontinuum includes the path of the vehicle. Therefore, the images wouldbe aligned in a dimension consistent with the path of the vehicle. Acomposite image of the subject is generated from the aligned portions ofthe two or more images.

In some embodiments, positional data is recorded that is descriptive ofa respective location of each of the two or more images. The positionaldata can include, for example, latitude and longitude coordinates andcan be used to associate the composite image with a particular portionof the subject. It is also within the scope of the invention to recordthe altitude of a camera used to capture the image data and therebyapproximating the altitude of the subject matter of the image data.

In some embodiments, a camera will be maintained approximatelyorthogonal to the subject captured and therefore, those embodiments willmaintain the camera at an angle about between 75° and 105° in relationto the subject matter.

In another aspect, some embodiments can include various overlays ofinformation on top of the continuum of image data. Overlays can includeone or more of: metadata; data related to a composite image; andgeospatial information.

BRIEF DESCRIPTION OF THE DRAWINGS

As presented herein, various embodiments of the present invention willbe described, followed by some specific examples of various componentsthat can be utilized to implement the embodiments. The followingdrawings facilitate the description of some embodiments:

FIG. 1A illustrates basic geographic images captured from a camera atdisparate points along a continuum and aligned with post processing.

FIG. 1B illustrates an alignment of portions of two or more imagescaptured from disparate points along a continuum to form a compositeimage.

FIG. 2 illustrates multiple images captured from disparate points alonga continuum and specific vertical slices that can be selected to createa composite image.

FIG. 3 illustrates a block diagram of exemplary lens distortion factorsand a preferred area of lens capture that can be selected in someembodiments of the present invention.

FIG. 4 illustrates disparate points on a continuum with an orientationvector associated with each respective point.

FIG. 5 illustrates a canyon formed by two continuums of image dataaccording to the present invention.

FIG. 6 illustrates apparatus that can be used to implement someembodiments of the present invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The present invention provides methods and apparatus for capturing imagedata from multiple disparate points along a continuum and assembling theimage data into a composite image. In some embodiments, the compositeimage can include its own continuum of two dimensional image data. Someembodiments can also include storing the captured image data withpositional data related to the captured image data. The positional datacan include, or be derived, for example, from latitude and longitudecoordinates of the position of a camera used to capture the image dataas well as an orientation of the camera.

Definitions

As used herein, Video DriveBy™ refers to street level video datacaptured in multiple angles encompassing a 360° view.

As used herein, Video FlyBy™ refers to Aerial/Satellite oblique(angular) view images with polygon line views.

As used herein, RibbonView™ refers to a film strip like view ofproperties which stands up at approximately 90° from a flat or angledaerial/satellite ortho image map and provides direct-on front images ofproperties to be displayed.

As used herein, Flash Viewer (Streaming Video) refers to directstreaming of video to an online user via a web browser.

Methods

Referring now to FIG. 1A, three images 101-103 are illustrated. Each ofthe images 101-103 is reproduced from image data captured from adisparate point on a continuum. As illustrated, a composite image 100 isformed by aligning two or more of the image data sets. Unlike stitchingprocesses previously known, the present invention creates a compositethrough alignment of portions of data from more than one data set.Alignment can be accomplished in image data processing. Using image dataprocessing, the images 101-103 are aligned to form a composite image100. The composite image 100 is essentially two dimensional image dataarranged as a second continuum, or ribbon. The second continuum includesongoing image data 101-103 captured from the points defining the firstcontinuum.

In some particular embodiments, the series of points of image capture inthe first continuum includes positions of a vehicle carrying an imagecapture device, such as a camera, as the vehicle traverses a pathproximate to a geographic area. The camera is positioned to captureimage data of the geographic area. Image data 101-103 is periodicallycaptured as the vehicle traverses the path. The motion of the vehicle,combined with the periodic capture of image data 101-103, therebyresults in image data 101-103 being captured from disparate points alongthe first continuum.

A preferred embodiment includes capture of image data with a cameramaintained orthogonal to the subject. Orthogonal capture of the imagedata provides consistency for subsequent composite of portions of theimage data captured. Therefore, data captured at an angle of betweenabout 75° and 105° can provide data most easily assembled into acontinuum of data. However, other angles may be used to create differenteffects.

During image data processing, some or all of the images are aligned toform a composite image in the form of a continuous pictorialrepresentation of the geographic area. One commercial embodiment of acontinuous pictorial representation includes RibbonView™ by Real DataSystems. RibbonView™ correlates a ribbon of geographic image data withgeospatial designations to facilitate identification of a particulargeographic area, as discussed more fully below. In various embodiments,image capture processing can be accomplished in real time or via postimage capture processing.

Referring now to FIG. 1B, in some embodiments, select portions 104-107of two or more sets of captured image data are aligned to generate thecomposite image 108. As illustrated, some preferred embodiments includevertical slices of data 104-107 aligned in a horizontal plane to formthe composite image 108. Unlike a traditional photograph taken with amacro lens, according to the present invention, the length of ahorizontal plane defining a composite image 108 is only limited by thelength of a continuum along which points are defined and from whichimage data 104-107 is captured.

The use of only slices of data 104-107 from any particular capturedimage provides for a higher quality image 108. The quality is increased,for example, when a temporary obstruction, such as a passing car, personor animal, captured in one image data set, is only represented in a thinslice of a continuous ribbon 104-107 data. In addition, alignment ofmultiple thin slices of image data 104-107 is facilitated from theperspective of which aberrations typical human sensory is capable ofdistinguishing. Numerous thin slices 104-107 are perceived as a smoothertransition across the horizontal plane defining the composite image 108removing a choppiness that may be experienced with the use of largerdata images.

The present invention can include a uniform width of each slice of data104-107 or a variable width. The width of a particular slice may vary,for example, upon one or more of the velocity of a vehicle from whichimage data sets are captured, the sample rate of a camera used tocapture an image data set 101-103, the resolution of a picturecomprising an image data set 101-103 and the path of a camera. A highresolution image generated by a 2.1 mega pixel camera may have a 1600 by1200 resolution and allow for a thinner slice 104-107 that includes awidth of between about 5 to 100 pixels of an image data set. Embodimentswith very high quality can include a slice 104-107 of between 1 to 2pixels.

From a different perspective, some embodiments can include a slice104-107 of an image data set 101-103 that includes a percentage of theimage data set 101-103. Accordingly, some preferred embodiments caninclude a slice 104-107 of between about 5% to about 12% of an imagedata set. Other preferred embodiments can include up to about 50% of animage data set. However, it should be understood that some embodimentsinclude a slice 104-107 that constitutes any fraction of the completeimage data set.

Referring now to FIG. 2, three sets of image data 201-203 areillustrated, wherein each set of image data 201-203 represents datacaptured from a disparate point on a continuum proximate to thelandscape captured. A direction of travel along the continuum 200 isalso illustrated. Each data set 201-203 is divided into multiple slicesof image data 204-209. A composite image 210 is generated from themultiple slices 204-209, and in the exemplary case at hand, additionalslices from additional image data sets.

It should be noted that although preferred embodiments may utilizevertical rectangular slices 204-209, the scope of the present inventionis not limited by which portion of an image data set 201-203 is utilizedto generate a composite image 210. Therefore, it is within the scope ofthe invention to use any portion of any image data set 201-203 togenerate a composite image. Accordingly, slices of an image 201-203other than vertical slices 204-209 may be apportioned and combined intoa composite image 210. Slices may therefore include a slice angled at60° or 75°, or other angle conducive to a particular application. Inaddition, it is also within the scope of the present invention toutilize irregular shaped portions of two or more image data sets 201-203to generate a composite image 210.

In some embodiments, a database or other data processing mechanism, cantrack each portion or slice 204-208 utilized to construct a continuumand associate the slice 204-208 with an original image 201-203 fromwhich the slice 204-208 is derived. User interactive devices can executethe retrieval of an entire original image 201-203 or a series oforiginal images 201-203 upon request. In some instances, one or moreoriginal images 201-203 may provide detail not included in the compositeimage 208.

Referring now to FIG. 3, in some embodiments, a selected portion 303 ofan image data set 300 may be determined by the physical attributes ofthe equipment used to capture an image data set. For example, a typicalcamera lens can impart some distortion to an image data set, asrepresented in the illustration by an elongated portion 301 and acompressed portion 302. Utilizing only a portion 302 of an image dataset 300, such as, for example, a center portion vertical slice 303, canminimize the effect of distortion introduced by a lens, or other source,to a composite image 108. Distortion is minimized when the compositeimage 108 is made to include those portions of the image data setcorresponding with the center of the lens 303. Specific embodiments maylikewise account for other aberrations that may be present in a set ofimage data.

In another aspect of the invention, and referring now to FIG. 4,positional data descriptive of a location of the subject matter of animage can also be generated by the image data recording device.Positional data can include any data indicative of where the subjectmatter of an image is located. Some preferred embodiments can includeCartesian coordinates that are context sensitive according to themechanism used to generate coordinate data.

For example, an image recording device, such as a camera, can beassociated with a device for recording a global position, such as aglobal positioning system (GPS) device or other equipment. Time stampsassociated with image data and time stamps associated with the GPS datacan be utilized to correlate the GPS location data with image datarecorded by the camera.

In still another aspect, in some embodiments, an altimeter can be usedto record an altitude from which a camera records image data sets. Thealtitude data can be associated with an image data set, for example,metadata correlated with the image data set. Such embodiments cantherefore include recordation of a latitude, longitude and altitudecoordinate for a given image data set. In addition, it is also withinthe scope of this invention to record a time of generation of an imagedata set and a depth of focus for an image data set.

According to the present invention, geospatial data, such as latitudeand longitude coordinates, can be generated by the GPS and stored withimage data generated by the camera. In some embodiments, GPS data can betime stamped and collected once every second. However, in someinstances, GPS reception can be interrupted, depending upon locationrelative to large object, such as multistory buildings, or cold cover.Therefore, some additional embodiments can include an accelerometer forrecording motion associated with a camera and a GPS unit operativelyattached to the camera.

Data from the accelerometer can indicate movement of the camera. Someaccelerometers, such as micro electromechanical system (MEMs)accelerometers can easily be incorporated into a camera system assemblyattached to a vehicle. Use of multiple MEM accelerometers positioned tomeasure movement in four or more directions along an x-axis, y-axis, andz-axis in relation to a camera can also be used to calculate directionof movement. The accelerometer can therefore be used to extrapolate acurrent position of the camera, based upon a last set of GPS geospatialdata recorded.

Geospatial data can be used to indicate an initial geographic position.A change in geospatial data can be additionally utilized to indicatevelocity and direction of image data set capture. Accelerometer data mayalso be used to indicate a velocity and direction of image data setcapture. Accelerometer data may also be used to indicate a velocity anddirection of camera movement. Calculations of time elapsed at theindicated velocity (such as for example, the Kalman Filter) can yield acalculated position at a time of image capture, even if the time ofimage capture is between GPS readings.

For example, one standard can include tracking a camera position with aGPS unit that records location at a rate of once per second. The cameracan record image data at a faster rate than once per second, such as,for example, one of: 12 images per second, 24 images per second or 29.97images per second. An initial camera position can be recorded whichcorrelates with a GPS reading, subsequent image data capture will occurin between the one second GPS reading interval. The camera position canbe determined with a calculation based upon the velocity of cameramovement supplied by the accelerometer and time elapsed since a previousGPS reading.

Still other embodiments can utilize optical flow methodology and visualodometry to facilitate calculations of a camera position and thevelocity of a vehicle or person from which a series of image data setsare captured. Visual odometry can be accomplished with a singleomni-directional camera or with stereo cameras, and is based uponprocessing which tracks the position of salient features in a series offeature sets and calculates movement based upon the relative positionsof the features. In some embodiments, camera based simultaneouslocalization and mapping (SLAM) of visual image data can also be used tofacilitate calculations of velocity of a change in position of a cameraused to capture image data sets. Typically, the velocity will bedirectly tied to the motion of a vehicle to which the camera is mounted,or a person carrying a camera rig.

Orientation of a camera can include a direction of image capturerecorded by the camera. Orientation can be designated, for example, inrelation to the cardinal directions, i.e. north, south, east and west.Any means available to record such a designation, such as an electroniccompass, is within the scope of the present invention. However, it maybe desirable to include a means to record the orientation with a greaterdegree of accuracy than is typically available through the use of anelectronic compass.

Therefore, in some embodiments, orientation can be determined accordingto a fixed position of a camera in relation to the direction of travelof a vehicle (or person) used to transport the camera. For example, aplurality of cameras can be fixedly attached to a vehicle capturingVideo DriveBy™ data. Each camera therefore maintains a constantdirection of image capture in relation to the heading of the vehicle.Mechanics of the camera, such as, for example, lens parameters andshutter speed, can indicate a depth of field during camera image datacapture. Some embodiments can also include simultaneously capturingmultiple image data sets and correlating two or more of the image datasets. Correlation can be accomplished via a time stamp or otherchronological or synchronous.

The position of a camera can be combined with a direction of imagecapture and the depth of field of the camera, to determine a location ofimage data captured by the camera at a particular instance in time. Thepresent invention can also include apparatus for utilizing echo locationto determine a distance of an object from a camera capturing an imagedata set and storing a correlated distance with the image data set. Forexample, radar data can be correlated with a camera image data set tocalculate the location of various objects captured by the camera. A timestamp can also be combined with data to quantify a location for aparticular image formed by the captured image data.

In some embodiments of the present invention data used to calculate alocation of an image is stored in a metadata file space associated withthe image data. For example, some embodiments can store metadata in theexchangeable image file format (EXIF), TIFFTAGS or International PressTelecommunication Council (IPTC) formats. Image data may be stored, forexample in JPEG or TIFF formats. However, other metadata formats canalso be used. Typically, due to the size of data files that aregenerated during capture of Video DriveBy™ data, the image data andmetafile data are stored on an external data storage device, such as ahard disk drive operatively attached to the camera. However, in someembodiments, the data can be stored in the camera.

As discussed above, the metadata can include data descriptive of some orall of: date and time; camera settings such aperture, shutter speed andfocal length; geospatial data from a GPS receiver unit; accelerometerdata; inertial guidance system data; camera orientation; and camerafixed position related to vehicle travel.

Referring now to FIG. 5, an illustration of some RibbonView™ embodimentsof the present invention is illustrated. The RibbonView™ illustrated501-502 includes strips or ribbons of two-dimensional image data. Eachstrip 501-502 is generated via the capture of image data along acontinuum 503. As illustrated, the continuum 503 generally follows theroadway in the center of the illustration. Each composite image 501-502is generated through the alignment of a portion of multiple imagescaptured as data from disparate points along the continuum 503. In somepreferred embodiments, the present invention delivers two ribbons ofcomposite images 501-502 with different views captured along thecontinuum. Each view is generated by a camera with a separateorientation.

It will be apparent to those schooled in the art that the length of acomposite image generated according to the present invention is limitedonly by the ability to capture image data from additional points on acontinuum and store the captured image data for post processing. Thepost processing allows for the alignment of portions of the image datacompiled into a composite two-dimensional view that can continue so longas additional image data is made available to be added to it.

Apparatus

The teachings of the present invention may be implemented with anyapparatus capable of embodying the innovative concepts described herein.Image capture can be accomplished, for example, via a digital cameracapable of capturing 12 or more images per second. In addition, FIG. 6illustrates a controller 600 that may be utilized to implement someembodiments of the present invention. The controller 600 comprises aprocessor unit 610, such as one or more processors, coupled to acommunication device 620 configured to communicate via a communicationnetwork (not shown in FIG. 6). The communication device 620 may be usedto communicate, for example, with one or more online devices, such as apersonal computer, laptop or a handheld device.

The processor 610 is also in communication with a storage device 630.The storage device 630 may comprise any appropriate information storagedevice, including combinations of magnetic storage devices (e.g.,magnetic tape and hard disk drives), optical storage devices, and/orsemiconductor memory devices such as Random Access Memory (RAM) devicesand Read Only Memory (ROM) devices.

The storage device 630 can store a program 640 for controlling theprocessor 610. The processor 610 performs instructions of the program640, and thereby operates in accordance with the present invention. Theprocessor 610 may also cause the communication device 620 to transmitinformation, including, in some instances, control commands to operateapparatus to implement the processes described above. The storage device630 can additionally store related data in a database 630A and database630B, as needed.

CONCLUSION

A number of embodiments of the present invention have been described.Nevertheless, it will be understood that various modifications may bemade without departing from the spirit and scope of the invention. Forexample, various methods or equipment may be used to implement theprocess steps described herein or to create a device according to theinventive concepts provided above and further described in the claims.In addition, various integration of components, as well as software andfirmware can be implemented. Accordingly, other embodiments are withinthe scope of the following claims.

1. Apparatus for processing a continuum of image data, the apparatuscomprising: a computer server comprising a processor and a storagedevice; and executable software stored on the storage device andexecutable on demand, the software operative with the processor to causethe server to: capture two or more images of a subject, wherein the twoor more images are captured from disparate points on a continuum; alignportions of the two or more images in a dimension consistent with thecontinuum; and generate a composite image of the subject comprising thealigned portions of the two or more images, wherein the software isadditionally operative to: record positional data descriptive of arespective location of each of the two or more images; and associate thecomposite image with a particular portion of the subject based upon thepositional data.
 2. The apparatus of claim 1 wherein the subjectcomprises a geographic area and the positional data comprises ageospatial designation.
 3. The apparatus of claim 2 wherein the softwareis additionally operative to generate the positional data based uponlatitude and longitude coordinates of a camera capturing the two or moreimages and data descriptive of the camera orientation while the cameracaptured the two or more images.
 4. The apparatus of claim 2 wherein thedifferent points in a continuum comprise positions of a vehicletraveling on a roadway.
 5. The apparatus of claim 2 wherein each of thetwo or more images are captured with a camera at a position aboutorthogonal to the subject matter.
 6. The apparatus of claim 5 whereinthe camera is at an angle of between about 85.degrees and about 95degrees relative to the subject matter.
 7. The apparatus of claim 5additionally comprising an altimeter and wherein the software isadditionally operative to record data indicative of an altitude of thecamera at the instance of image data set generation.
 8. The apparatus ofclaim 2, wherein the composite image comprises a single vertical sliceof each of the two or more images.
 9. The apparatus of claim 8 whereinthe images are captured by a camera mounted to a vehicle traveling inproximity to a subject matter comprising a geographic parcel and a widthof each vertical slice is based upon a speed of image capture of thecamera and the velocity of the vehicle travel.
 10. The apparatus ofclaim 2 wherein the software is additionally operative to combine thecomposite image with an aerial view image of the geographic area. 11.The apparatus of claim 10 wherein the aerial view comprises aperpendicular view of the geographic area.
 12. The apparatus of claim 11wherein the aerial view image comprises an oblique view of thegeographic area.
 13. The apparatus of claim 1 wherein the software isadditionally operative to overlay metadata on the composite imagedescriptive of the composite image.
 14. The apparatus of claim 1 whereinthe software is additionally operative to overlay links to additionaldata related to the composite image.
 15. The apparatus of claim 14wherein the additional data is related to the composite image based uponthe positional data.
 16. The apparatus of claim 1 wherein the two ormore images are captured with a camera oriented in a direction aboutorthogonal to a line between two points on the continuum.
 17. Theapparatus of claim 16 wherein the camera is at an angle of between about75.degree. and about 105 degrees relative to the line.
 18. The apparatusof claim 1 additionally comprising a data file comprising the compositeimage and geospatial data descriptive of the composite image.
 19. Theapparatus of claim 18 wherein the structure of the data file comprisesmultiple digital images and geospatial data stored between at least someof the multiple digital images.